Engineers use compression (also called source coding or source encoding) to reduce the bitrate of digital video. Compression decreases the cost of storing and transmitting video information by converting the information into a lower bitrate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec” is an encoder/decoder system.
Over the last two decades, various video codec standards have been adopted, including the H.261, H.262 (MPEG-2) and H.263 standards and the MPEG-1 and MPEG-4 standards. More recently, the H.264 standard (sometimes referred to as AVC or 14496-10) and VC-1 standard have been adopted. For additional details, see representative versions of the respective standards. A video codec standard typically defines options for the syntax of an encoded video bitstream, detailing parameters that must be in the bitstream for a video sequence when particular features are used in encoding and decoding. In many cases, a video codec standard also provides details about the decoding operations a decoder should perform to achieve correct results in decoding.
A basic goal of compression is to provide good rate-distortion performance. So, for a particular bitrate, an encoder attempts to provide the highest quality of video. Or, for a particular level of quality/fidelity to the original video, an encoder attempts to provide the lowest bitrate encoded video. In practice, depending on the use scenario, considerations such as encoding time, encoding complexity, encoding resources, decoding time, decoding complexity, decoding resources, overall delay, and/or smoothness in playback also affect decisions made during encoding and decoding.
For example, consider use scenarios such as video playback from storage, video playback from encoded data streamed over a network connection, and video transcoding (from one bitrate to another bitrate, or one standard to another standard). At the encoder side, such applications may permit off-line encoding that is not at all time-sensitive. Therefore, an encoder can increase encoding time and increase resources used during encoding to find the most efficient way to compress video, and thereby improve rate-distortion performance. If a small amount of delay (e.g., 3-10 seconds at startup) is also acceptable at the decoder side, the encoder can further improve rate-distortion performance, e.g., by exploiting inter-picture dependencies from pictures farther ahead in a sequence.
For most video playback scenarios, at the decoder side, delay of 3-10 seconds or more is acceptable before playback begins, so long as playback is uninterrupted after it starts. Subject to this smooth playback constraint, the decoder is free to devote additional resources for multi-threaded decoding or multi-processor decoding. In fact, in some cases, a decoder can increase latency so as to allow more efficient utilization of decoding resources.
On the other hand, consider use scenarios such as remote desktop conferencing, surveillance video, video telephony and other real-time communication scenarios. Such applications are time-sensitive. Low latency between recording of input pictures and playback of output pictures is a key factor in performance. When encoding/decoding tools adapted for non-real-time communication are applied in real-time communication scenarios, overall latency is often unacceptably high. The delays that these tools introduce during encoding and decoding may improve performance for regular video playback, but they disrupt real-time communication.